The present invention relates to color grading. More specifically, the present invention relates to methods and apparatus for color grading and gamut matching of displayed images to other media.
Throughout the years, movie makers have often tried to tell stories involving make-believe creatures, far away places, and fantastic things. To do so, they have often relied on animation techniques to bring the make-believe to “life.” Two of the major paths in animation have traditionally included, drawing-based animation techniques and stop motion animation techniques.
Drawing-based animation techniques were refined in the twentieth century, by movie makers such as Walt Disney and used in movies such as “Snow White and the Seven Dwarfs” (1937) and “Fantasia” (1940). This animation technique typically required artists to hand-draw (or paint) animated images onto a transparent media or cels. After painting, each cel would then be captured or recorded onto film as one or more frames in a movie.
Stop motion-based animation techniques typically required the construction of miniature sets, props, and characters. The filmmakers would construct the sets, add props, and position the miniature characters in a pose. After the animator was happy with how everything was arranged, one or more frames of film would be taken of that specific arrangement. Stop motion animation techniques were developed by movie makers such as Willis O'Brien for movies such as “King Kong” (1933). Subsequently, these techniques were refined by animators such as Ray Harryhausen for movies including “Mighty Joe Young” (1948) and Clash Of The Titans (1981).
With the wide-spread availability of computers in the later part of the twentieth century, animators began to rely upon computers to assist in the animation process. This included using computers to facilitate drawing-based animation, for example, by painting images, by generating in-between images (“tweening”), and the like. This also included using computers to augment stop motion animation techniques. For example, physical models could be represented by virtual models in computer memory, and manipulated.
One of the pioneering companies in the computer-aided animation (CA) industry was Pixar. Pixar is more widely known as Pixar Animation Studios, the creators of animated features such as “Toy Story” (1995) and “Toy Story 2” (1999), “A Bugs Life” (1998), “Monsters, Inc.” (2001), “Finding Nemo” (2003), “The Incredibles” (2004), “Cars” (2006) and others. In addition to creating animated features, Pixar developed computing platforms specially designed for CA, and CA software now known as RenderMan®. RenderMan® was particularly well received in the animation industry and recognized with two Academy Awards®. The RenderMan® software included a “rendering engine” that “rendered” or converted geometric and/or mathematical descriptions of objects and forms a two dimensional image.
In film making, after the live-action images are shot, or after animated images are rendered, one operation typically performed on such images is known as color grading or color timing. With color grading, the colors of the image are typically modified until a certain “look” is desired. For example, a live action scene may be filmed at noon, and with color grading, a user could make the images appear as though it were night by increasing the cooler colors (e.g. blues) in the images. As another example, live action scenes may be filmed out of order throughout a day, and by performing color grading, the colors of the scenes could be adjusted until the light in the proper sequence of scenes is “correct.” Additionally, color grading may be performed on animated features to produce the same results described above. Color grading of the images in a feature is typically a time-consuming and labor intensive process that requires exacting control of the colors in the images. Additionally, color grading is typically an artistic process performed by a skilled artisan known as a colorist.
After frames of a film are color graded and the Director is satisfied with the appearance of the images on the computer display, the images may be transferred onto film media for distribution into theaters, onto paper or plastic media, or the like. As discussed in the above-mentioned patent application, one issue that arises when transferring images computed and displayed on a computer display, is that the images will look different on the different types of media, e.g. film in the theater, DVDs on home theater systems, magazines, etc. Some reasons for this include that the color gamut and color reproduction of a user display and that the color gamut and color reproduction of different target media are often very different.
With regards to color reproduction (typically non-linear) response of film media, the inventors recognize that characterization is typically independent of the rendered image. For example, to characterize the color response, one or more images having ramped color densities are recorded onto film stock with a film recorder, and the resultant color densities of the film are measured. Then, based upon the known color density output and the corresponding measured color density, the color reproduction of the film media is determined.
With regards to matching colors displayed on the display to colors in the color gamut of film media, the inventors now recognize that it is desirable to limit the amount of automatic gamut matching so as to reduce unintended side effects.
Automatic approaches are typically used to map colors outside a first color gamut (out-of-gamut) to fit within a color gamut (in-gamut). Often these approaches are characterized by rendering intent. Four common automatic methods for performing the rendering intent are known as Saturation, Relative and Absolute Colorimetric, and Perceptual intents.
For motion pictures, a problem with implementations of saturation intents, where the vividness of pure colors are desired to be preserved, is that out-of-gamut colors typically, undesirably change in hue. For example, a yellow flower may be turned into an orange flower as a result of such a color transformation. Accordingly, saturation intents approaches to out-of-gamut colors are not typically used for motion pictures (live action, or animation).
A problem with implementations of perceptual mapping intents, where all colors are remapped to in-gamut colors, is that all colors tend to change. Additionally, the dynamic range of colors is reduced. As a result, the colors of a modified image may not appear as vivid or saturated as was originally intended. Accordingly, perceptual intents approaches to out-of-gamut colors are not typically used for motion pictures (live action, or animation) as the resulting images may appear dull.
Implementations of automatic colorimetric (relative or absolute) intents are more often used in the photographic and motion picture industries. This is because with these techniques, pre-existing in-gamut colors are maintained (absolutely or relatively with respect to a defined white point), while out-of-gamut colors are pulled in-gamut. Typically, such colorimetric intents approaches move out-of-gamut colors towards the neutral axis until an in-gamut color is reached.
FIG. 1 illustrates an example of automatically moving out-of-gamut colors to in-gamut colors, independent of color grading judgments. Illustrated in FIG. 1 is a two dimensional portion of a color chart 100 of blue values plotted against green values. In this example, red values point into the page. Color chart includes a representation of a color gamut 110 for a first media (e.g. a monitor) and a representation of a color gamut 120 for a second media (e.g. film) in the color space of the monitor (RGB). Outlined in FIG. 1 are regions 130 representing colors within color gamut 110, but out-of-gamut with respect to color gamut 120. Additionally, a line 150 represents a projection of a neutral axis upon the blue/green plane. As illustrated, in the color space representation 160, the neutral axis 150 actually runs equidistant between the red, green, and blue axes, from black to white.
In the example in FIG. 1, a color 180 is within region 130, and is out-of-gamut with respect to color gamut 120. With a typical gamut remap using absolute colorimetric intents, color 180 is moved towards neutral axis 150 until an in-gamut color is obtained. In this example, the new color is indicated by color 190. As an example, color 180 has blue, green coordinates of (0.90, 0.20), and color 190 has blue green coordinates of (0.60, 0.40). Accordingly, color 180 is more much more bluish than color 190. Other colors, such as color 193 may also be moved towards the neutral axis, to color 195, as illustrated.
As can be seen in representation 160, a more accurate movement of color 180 towards neutral axis 150, also adds a red value to color 180. Thus, for example, color 180 may have red, green, blue (RGB) coordinates of (0, 0.20, 0.90), and color 190 may have RGB coordinates of (0.20, 0.40, 0.60). Accordingly, color 180 which is bluish to begin with, is remapped to color 190 that is more pinkish. Thus, even if a colorist specifies a dark blue color 180, the gamut remap process will automatically turn the dark blue into a pinkish blue.
A significant drawback to using automatic gamut matching operations is that it often ruins the painstaking color grading performed by the colorist. More specifically, as stated above, the color grading of images is a function that is typically a time and labor intensive process. Further, this process is typically performed painstakingly by a color expert/artist who determines where each color should be placed. Accordingly, using automatic gamut matching processes after a color grading process will ruin the color grading for the images. As was illustrated in FIG. 1, working within color gamut 110 of a computer monitor, the color grader may like color 180 and select its value during the color grading process. However, as a result of the automatic gamut matching process, color 180 is automatically remapped to color 190. As an example, a deep blue of color 180 (e.g. (0, 0.90, 0.20) by the color grader may be automatically moved to a pinkish blue of color 190 (e.g. (0.20, 0.60, 0.40), as a result of the automatic gamut matching. Thus as can be seen, the inventors have determined that such automatic gamut matching is disadvantageous with regards to color graded images, and that the automatic gaming matching often “fights” the intentions of the color grader.
One technique sometimes used by film makers is limiting the gamut of a digital intermediate to the film gamut throughout the workflow. For example, the universe of colors on the digital display will be an intersection of the film gamut and the display gamut. In such a system, there is no color gamut matching problem when performing film-out, thus gamut matching is not needed when printing to film.
One drawback to such a system is that the film makers are visually restricted to working in the narrow color gamut (i.e. the intersection of the film gamut and the display gamut). Accordingly, if the film is transferred to DVD for a home theater system or projected digitally on a digital projector, the colorist will not be able to preview the grading of the resulting images. Another drawback is, there is no way to increase the color gamut of the film, without repeating the entire digital workflow. As an example, after a film is mastered for film, if the producer wanted to show their film on a digital projection system, the entire digital workflow would have to be repeated. This is because the film was not designed for the gamut of the digital projection system, which is typically wider in some respects compared to film. In such cases, to re-master for a digital projection system, special effects, live action shots, etc. would most likely have to be regraded, to make use of the full gamut of the digital projection system.
In light of the above, what is desired are more efficient ways for performing color grading and gamut matching, without the drawbacks described above.